Remove custom celery routing and exchange #2193
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
Due to the implementation of async SJS requests, we established a mechanism for directly establishing routes that would stay on a single worker. This led to complex code and also high latency when we pinged each worker to determine if they were available. With recent changes removing SJS in favor of a synchronous geoprocessing call, we can now cull these custom routes and exchanges, both simplifying the code and increasing the speed of submitting a new job.
Connects #2117
Fixes #1903
Fixes #1894
Fixes #1535
Notes
This reduces the HTTP response time for all celery submit endpoints considerably. Locally, they return in ~250ms, as opposed to seconds on staging/production.
Testing Instructions
First round of testing is to ensure that all geoprocessing tasks work (except Mapshed which is still being refactored - however, the mapshed jobs should still be submitted correctly, the streams will fail).
/land, /soil, /tr55, /rwd
, etc) respond in milliseconds and not secondsSecond round of testing is to make sure that the work is still adequately shared between workers and that stacks of another color don't process jobs submitted for another stack color.
Follow these instructions sequentially, as the steps build on each other.
Test multiple workers
./scripts/debugcelery.sh
to launch a worker thread on the Black stackNotice the default key is
task.Black
in the outputTest multiple stack colors
vagrant ssh worker
Black
->Blue
sudo vim /etc/mmw.d/env/MMW_STACK_COLOR
/opt/app
, launch a third celery worker, this one will start as if on a Blue stack.Notice the default key is
task.Blue
in the outputTest switching stack task submission
settings_base.py
and setSTACK_COLOR = 'Blue'
settings_base.py
and/etc/mmw.d/env/MMW_STACK_COLOR